Image enhancement and data loss recovery using wavelet transforms

ABSTRACT

Methods and systems for image and video enhancement and data-loss recovery using wavelet transforms are disclosed. A wavelet transformed image is defined as multiple sections, where one section is the original image, and coefficients are estimated for each other section using information obtained from that section. An inverse wavelet transform on the wavelet transformed image obtains an enlarged image. For obtaining a high resolution video frame, one section is the original video frame, and coefficients are estimated for each other section using information from a lower level wavelet transform of a corresponding section of a past or future frame of the video sequence. For recovery of data lost from transmitted original data, the locality of the lost data is determined in lower and higher levels of a wavelet transform tree, and information at the locality in the lower and higher levels is used to estimate the lost data.

FIELD OF THE INVENTION

[0001] The present invention relates to techniques for the enhancementand recovery of image data and other data, and more particularly to theuse of wavelet transforms for image data enhancement and data recovery.

BACKGROUND OF THE INVENTION

[0002] One application in the manipulation of data used by computer andelectronic devices is the compression and decompression of data. Storagespace for data in memory devices is limited in many circumstances, sothat data compression techniques are often used to reduce the amount ofstorage space that is needed for an image, a message, or other block ofdata. Once compressed and stored, the compressed data is eventuallydecompressed into its uncompressed, original form using an algorithm ortechnique complementary to the compression technique. Some types ofcompression are known as lossy, where some data is lost in thecompression and decompression process. However, in many applications,such as image compression, the lost data typically does not make anoticeable or practical difference in the final use or application ofthe data.

[0003] One lossy compression technique that is used often in recentyears is known as wavelet-based compression. In this type ofcompression, a wavelet transform is used to reduce the amount of datawith little noticeable loss. One type of wavelet transform that can beperformed using digital processors and circuits is the Discrete WaveletTransform (DWT), which uses discrete samples of a continuous wavelet,and can be similar to a filtering technique with discrete coefficients.The DWT can be tuned to a particular application or input, allowing itin many cases to be more useful for applications such as imagecompression or enhancement than other transforms such as the discretecosine transform (DCT) or averaging filters. For example, the JPEG2000still image compression standard is wavelet-based.

[0004] Data enhancement is useful and desirable in a wide variety ofcomputer-and electronic-based applications. A data enhancementapplication that can be related to data compression is imageenlargement. Instead of decompressing a compressed image, an original,uncompressed image is enlarged to a greater size. This can be useful inmany applications; for example, an original image in a lower resolutionmay be too small to fit a particular screen size and preferably shouldbe enlarged. Also, super resolution images can be created from smallerimages for prints or photographic quality pictures. Video images orstreams can be enhanced by increasing the resolution of the particularvideo images or by estimating entire frames between existing frames toincrease the smoothness of motion perceived in the visual presentation.

[0005] When enlarging images, different techniques can be employed. Someprior methods include duplicating pixels to achieve the higherresolution, or using a bi-linear interpolation or other averagingtechnique. However, these techniques typically result in images of poorquality, having a “blocky” appearance or rough contours.

[0006] Other enlargement techniques may use wavelet transforms insimilar ways to wavelet-based compression. For example, one technique ofthe prior art constructs virtual DWT sub-bands from an image withoutperforming the DWT and applies an Inverse Discrete Wavelet Transform(IDWT) upon the virtual sub-bands, where the result of the IDWTrepresents an up-sampled (enlarged) version of the image. However, sincethe virtual sub-bands to be used with the IDWT do not exist, they aresimply zeroed. The result is a regenerated image that may be better thanother standard techniques but the regenerated image still typically haslow quality.

[0007] Data enhancement can also include recovering lost data incommunication channels. For example, data can get lost duringtransmission and reception, over any kind of communication channel. Therecovery of the lost data is important for reliable communications.However, existing techniques may not recover data as reliably as desiredfor image, video, and other kinds of data transmission.

SUMMARY OF INVENTION

[0008] A method and system for image enhancement and data-loss recoveryusing wavelet transforms is disclosed. In a first aspect, a method andsystem comprises defining a wavelet transformed image as a plurality ofsections, wherein one of the sections is the original image andestimating coefficients in estimated sections of the wavelet transformedimage. The coefficients are estimated for each particular estimatedsection using information obtained from that section. The method andsystem includes performing an inverse wavelet transform on the wavelettransformed image. The wavelet transformed image, including the originalimage and the estimated sections, to obtain the enlarged image.

[0009] In a second aspect, a method and system for creating highresolution video frames in a video sequence is disclosed: The method andsystem comprise defining a wavelet transformed frame as a plurality ofsections, wherein one of the sections is a present video frame andestimating coefficients in estimated sections of the wavelet transformedframe. The coefficients are estimated for each particular estimatedsection using information from a lower level wavelet transform of acorresponding section of a past or future frame of the video sequence.The method and system also includes performing an inverse wavelettransform on the wavelet transformed frame. The wavelet transformedframe includes the present video frame and the estimated sections, toobtain a high resolution video frame having a greater resolution thanthe present video frame.

[0010] In a third aspect, a method and system for recovering data lostfrom original data during transmission of the original data in acommunication channel is disclosed. The method and system comprisedetermining the locality of the lost data in a wavelet transform tree,the wavelet transform tree having been created from the original databefore transmission; determining the locality of the lost data in lowerand higher levels of the wavelet transform tree and in wavelet transformquadrants corresponding to the locality of the lost data; and usinginformation at the locality in the lower and higher levels of thewavelet transform tree to estimate the lost data.

[0011] The present invention provides several methods and apparatus forproviding enhanced, high resolution images and enhanced video sequences,and for recovering data lost during transmission. The techniquesdescribed allow images to be expanded in size and resolution with littleloss in quality, and allows video frames to be reconstituted to create asmoother video presentation. The data recovery techniques allow lostdata to be reliably reconstituted and allow the most important data tobe transmitted with little loss.

BRIEF DESCRIPTION OF THE DRAWINGS

[0012]FIG. 1 is a diagrammatic illustration of the transformation of anoriginal image into a one-level wavelet transformed image.

[0013]FIG. 2 is a diagrammatic illustration of the transformation of aportion of an original image into three levels using a wavelettransform.

[0014]FIG. 3 is a diagrammatic illustration of the original image andestimated wavelet coefficient quarters of the present invention beforeinverse discrete wavelet transformation.

[0015]FIG. 4 is a block diagram illustrating the present invention forenlarging a still image.

[0016]FIG. 5 is a flow diagram illustrating an iterative process of thepresent invention for creating a still image of enhanced resolution froman original image.

[0017]FIG. 6 is a diagrammatic illustration of the correlation ofcoefficients at different wavelet transform levels.

[0018]FIG. 7 is a diagrammatic illustration of the use in the presentinvention of quarter-specific image property information for coefficientestimation.

[0019]FIG. 8 is is a diagrammatic illustration of the estimation ofwavelet coefficients using averaging of corresponding coefficients.

[0020]FIG. 9 is a diagrammatic illustration of a sequence of videoimages and a technique of the present invention to estimate waveletcoefficients.

[0021]FIG. 10 is a block diagram illustrating the embodiment of thepresent invention for estimating wavelet coefficients in a videosequence.

[0022]FIG. 11 is a flow diagram illustrating a process of the presentinvention for estimating wavelet coefficients for a series of videoimages to provide enlarged video images or other enhanced video images.

[0023]FIG. 12 is a diagrammatic illustration of an example of therecovery of data lost in a communication channel.

[0024]FIG. 13 is a diagrammatic illustration of the relative importanceof the blocks resulting from wavelet transforms when recovering lostdata in the present invention.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

[0025] The present invention relates to techniques for the enhancementand recovery of image data and other data, and more particularly to theuse of wavelet transforms for image data enhancement and data recovery.The following description is presented to enable one of ordinary skillin the art to make and use the invention and is provided in the contextof a patent application and its requirements. Various modifications tothe preferred embodiment and the generic principles and featuresdescribed herein will be readily apparent to those skilled in the art.Thus, the present invention is not intended to be limited to theembodiments shown but is to be accorded the widest scope consistent withthe principles and features described herein.

[0026] Several embodiments and examples of the present invention aredescribed below. While particular applications and methods areexplained, it should be understood that the present invention can beused in a wide variety of other applications and with other techniqueswithin the scope of the present invention.

[0027] The present invention includes the use of wavelet transforms.Wavelet transforms have substantial advantages over conventional Fouriertransforms for analyzing nonlinear and non-stationary time series. Thesetransforms are used in a variety of applications, some of which includedata smoothing, data compression, and image reconstruction, among manyothers.

[0028] Wavelet transforms such as the Discrete Wavelet Transform (DWT)can process a signal to provide discrete coefficients, and many of thesecoefficients can be discarded to greatly reduce the amount ofinformation needed to describe the signal. One area that has benefitedthe most from this particular property of the wavelet transforms isimage processing. The DWT can be used to reduce the size of an imagewithout losing much of the resolution. For example, for a given image,the DWT of each row can be computed, and all the values in the DWT thatare less then a certain threshold can be discarded. Only those DWTcoefficients that are above the threshold are saved for each row. Whenthe original image is to be reconstructed, each row can be padded withas many zeros as the number of discarded coefficients, and the inverseDiscrete Wavelet Transform (IDWT) can be used to reconstruct each row ofthe original image. Or, the image can be analyzed at different frequencybands, and the original image reconstructed by using only thecoefficients that are of a particular band.

[0029]FIG. 1 illustrates the transformation of an original image 10 intoa one-level sub-sampled image 12. Wavelet transforms can decompose anoriginal image into sub-images, each sub-image representing a frequencysubset of the original image. Wavelet transforms sub-sample the image toprovide a successive decomposition of the original image into high- andlow-frequency components. One level of two dimensional wavelet transformcreates four sub-sampled separate quarters, each containing differentsets of information about the image. It is conventional to name the topleft quarter Low-Low (LL)—containing low frequency horizontal and lowfrequency vertical information; the top right quarter High-Horizontal(HH)—containing high frequency horizontal information; the bottom leftquarter High-Vertical (HV)—containing high frequency verticalinformation; and the bottom right quarter High-Diagonal (HD)—containinghigh frequency diagonal information. The level of transform is denotedby a number suffix following the two-letter code. For example, LL(1)refers to the first level of transform and denotes the top left cornerof the sub-sampled image 12.

[0030] Typically, wavelet transforms are performed for more than onelevel. FIG. 2 illustrates further transforms that have been performed onthe LL quarter of the sub-sampled image 12 to create additionalsub-sampled images. The second transform performed on the LL(1) quarterproduces four second level quarters within the LL(1) quarter which aresimilar to the first level quarters, where the second level quarters arelabeled as LL(2) (not shown), HH(2), HD(2), and HV(2). A third transformperformed on the LL(2) quarter produces four third level quarterslabeled as LL(3), HH(3), HD(3), and HV(3). Additional transforms can beperformed to create sub-sampled images at lower levels. A hierarchy ofsub-sampled images from wavelet transforms, such as the three levels oftransform shown in FIG. 2, is also known as a “wavelet transform tree.”

[0031]FIG. 3 is a diagrammatic illustration 16 of a technique used inthe present invention for providing images of increased resolution froman original image. The present invention takes the original image 10 andup-scales it using an inverse wavelet transform by assuming the originalimage 10 is the LL quarter LL(−1) of a sub-sampled, wavelet transformedimage of the up-scaled image for the next level. The other, remainingblocks, high frequency quadrants HH, HD, and HV, do not exist and areestimated as sub-bands: quarters HH(−1), HD(−1), and HV(−1), as detailedfurther below. The blocks for this original and first estimated levelcan be designated with (−1), since they are provided for transformationsin the opposite direction to a standard wavelet transform, whichprovides levels in a positive number direction and where the originalimage itself can be thought of as four quarters at level 0. The originalimage and these estimated quarters are then used in an inverse DWT(IDWT) to obtain an image having double the size and may result in anincrease in resolution of the original image in both x and y directions.The accuracy of the estimation technique determines the final quality ofthe image. This process can be repeated any number of times to obtainsuper-resolution images from the original image. For example, the higherresolution image obtained from the IDWT is again used as the LL quarterand the other three quarters are estimated, and the four quarters areinverse transformed to obtain an even higher resolution image.

[0032]FIG. 4 is a block diagram 20 illustrating the present inventionfor enlarging a still image. The block diagrams and flow diagramsillustrated herein are preferably implemented using software on anysuitable general-purpose computer or the like, having microprocessor,memory, and appropriate peripherals, where the software is implementedwith program instructions stored on a computer readable medium (memorydevice, CDROM or DVDROM, magnetic disk, etc.). The block diagrams andmethods can alternatively be implemented using hardware (logic gates,etc.) or a combination of hardware and software.

[0033] The original image 10 is input to a wavelet transformer 22. Thewavelet transformer 22 provides a number of wavelet transforms of theoriginal image, e.g. at levels 1, 2, 3, etc., which are then used in theestimation of the unknown quarters at the −1 level, as described below.The number of iterations of wavelet transform is not fixed, so that awavelet transform can be applied as many times as needed for the desiredamount of up-scaled resolution, depending on how many levels oftransform is to be used by the estimation block. Next, in the estimator24, HH, HD, and HV blocks having the same size as the original image areestimated using the coefficients of the known LL block (original image)and the coefficients in the lower levels of wavelet transforms of theoriginal image obtain by the transformer 22. At the first (original)level of estimation, the blocks are labeled HH(−1), HD(−1), and HV(−1).The original image is used as the LL(−1) sub-band of the level (−1)wavelet transformed image, as the dynamic range of wavelet coefficientsare different from the dynamic range of pixels in spatial domain, pixelsare scaled linearly according to the upper and lower limits of thedynamic ranges and the estimated blocks are provided as the appropriatesub-bands of the level (−1) wavelet transformed image. Next, the InverseDiscrete Wavelet Transformer 26 applies the IDWT to the original imageand estimated sub-bands. The output of the transformer 26 is an enlargedimage 28 having twice the size of the original image. If an even largeror higher resolution image is desired, the enlarged image 28 can be fedback to the estimator 24, since the lower level transforms are alreadyknown from the previous iteration, wavelet transformation is notnecessary for further enlargement. The process can be repeated furtherfor even higher resolution images.

[0034]FIG. 5 is a flow diagram 30 that illustrates an iterative processof the present invention for creating a still image of enhancedresolution from an original image. The process begins at step 32, and instep 34, a new level of the transform is started. If it is the firstiteration of this process, then the new level is the (−1) level, asexplained above. In step 36, the known image is placed in the LL(−x)quarter. The “known image” is the original image the first time throughthe process, and in later iterations is the image resulting from theimmediately previous transformation. The designation “x” is the currentlevel of transform, e.g., x is 1 in the first iteration.

[0035] In step 38, the estimation method is selected for the currentprocess. The estimation method can be one or more any of a plurality ofdifferent available types; one estimation method may be more efficientor require less computation than another method, while a differentestimation method may provide higher quality images, so that the mosteffective estimation method for a particular desired result can beselected based on criteria provided by the user or by using an analysismethod Some examples of estimation methods of the present invention aredescribed below.

[0036] In step 40, the accuracy required for the present transformationis determined. The level of accuracy can be based on how muchcomputation resources the user wishes to allocate, the time taken toperform the transforms, or other criteria of the user. In general, thegreater the number of transforms taken (to lower levels), the greaterthe accuracy obtainable. But, in this embodiment a coefficient in alower level corresponds to 4 coefficients in a next upper level.Similarly, each of these 4 coefficients corresponds to 4 coefficients inthe next upper level, making one coefficient corresponding to 16coefficients in two level above and 64 coefficients in the three levelabove. As this number increases according to the number of difference inlevel, if too many levels of transform is taken, the informationextracted from the lowest level coefficients doesn't help estimation ofthe wavelet coefficients, because they correspond to too manycoefficients representing a large spatial area.

[0037] In step 42, the number of sub-bands (levels of transform) to beused is selected This determines the accuracy of the estimation of aquarter; thus, the number of sub-bands is based upon the desiredaccuracy as determined in step 40.

[0038] In step 44, the HH(−x) quarter is estimated using HH levelcoefficients of the transform. In step 46, it is checked whether theaccuracy of the estimated HH quarter is sufficient. If not sufficient,in step 48, more sub-bands are provided, e.g., more coefficients fromother transform levels are provided. The additional sub-bands here canbe taken from transforms on the original image previously performed asdecided in steps 40 and 42. Alteratively, the transforms on the originalimage can be performed on the fly, as needed. Step 44 is then performedagain, where the HH(−x) quarter is estimated using the additionalsub-bands so that greater accuracy is achieved. This provision of moresub-bands for greater accuracy is performed repeatedly in a similarfashion until the accuracy is at the desired level at step 46. In nextstep 50, the HH(−x) block that was estimated in step 44 is placed in itsappropriate location for the eventual inverse transform, e.g., the HHblock is placed in the top right of the wavelet transformed image.

[0039] Steps 52-58 can be performed simultaneously to steps 44-50 orsequentially, as desired. The steps 52-58 are preferably similar tosteps 44-50 described above. In step 52, the HD(−x) quarter is estimatedusing HD coefficients of the transform. In step 54, it is checkedwhether the accuracy of the estimated HD quarter is sufficient. If not,in step 56, more sub-bands are provided and step 44 is performed againiteratively until the accuracy is at the desired level at step 54. Innext step 58, the HD(−x) block estimated in step 52 is placed in itsappropriate location for the eventual inverse transform, e.g., the HDblock is placed in the bottom right of the wavelet transformed image.

[0040] Steps 60-66 are similar to steps 44-50 and 52-58 and can beperformed simultaneously to steps 44-50 and 52-58 or sequentially, asdesired. In step 60, the HV(−x) quarter is estimated using HVcoefficients of the transform, and in step 62, it is checked whether theaccuracy of the estimated HD quarter is sufficient. If not, moresub-bands are provided in step 64 and step 60 is performed againiteratively until the accuracy is at the desired level at step 62. Innext step 66, the HV(−x) block estimated in step 60 is placed in itsappropriate location for the eventual inverse transform, e.g., the HVblock is placed in the bottom left of the wavelet transformed image.

[0041] After the steps 50, 58, and 66 are completed, step 70 isinitiated, in which the inverse wavelet transform (IDWT) is performed at(−x) with the known LL block and the estimated HH, HD, and HV blocks.The result of this inverse transform after the first iteration is animage double the size of the original image. In successive iterations,greater-sized images are produced by the inverse transform. After step70, therefore, the process checks in step 72 whether another level ofestimation and transform is to be performed. Thus, if an image ofgreater size is desired, another transformation is performed byreturning to step 34 to start a new level. At later levels (iterations),the quality of estimation is dependant on the quality of estimation atlower levels, which determines the overall quality of image expansion.If another level of transformation is not desired at step 72, then theprocess is complete at 74.

[0042] Estimation of Blocks

[0043] There are a number of different types of estimation and/orextrapolation methods of the present invention that can be used forsteps 44, 52, and 60 in the method 30 of FIG. 3 to provide coefficientsfor the unknown high frequency quarters. If the original image data isnot in the wavelet domain, a wavelet transform (such as DWT) of theoriginal image can be performed. The level of the transform that isperformed can be of any desired level; for example, a level 3 transformcan be performed to obtain two levels of transform of the originalimage. Once the original image data has been transformed to the desirednumber of wavelet transform levels (or is already compressed in thewavelet domain), then the necessary processing is applied to obtainwavelet coefficients from the transform levels, and then correlationbetween the coefficients in corresponding quarters at different levelscan be extracted using different mathematical models. FIG. 6 is adiagrammatic illustration 100 of this correlation between correspondingcoefficients at different transform levels in the wavelet domain. Onewavelet coefficient 102 at level 3, X(1)_(HD3), is a child of fourwavelet coefficients 104 in level 2, namely X(1 . . . 4)_(HD2)., atspatially corresponding locations of the same type of quarter, in thisexample the HD quarter. Similarly, the four wavelet coefficients 104 arechildren of sixteen wavelet coefficients 106 at level 1, namely X(1 . .. 16)_(HD1), in the same type of quarter of level 1 (HD in thisexample).

[0044] The class and type of correlation between the waveletcoefficients at different levels vary depending on the type of image.The general rule is that the absolute values of the wavelet coefficientsat corresponding spatial locations follow a pattern. Thus, once thepattern is determined, the absolute values of coefficients for unknownquarters at higher levels can be extrapolated or estimated based on thepattern determined from one or more lower levels of the transform.However, the signs of these coefficients may not necessarily follow thesame pattern. In such circumstances, signs can be estimated using asuitable probability function or statistical estimation technique, forexample edge information could be used while constructing thisprobability function; or the wavelet coefficients for this particularlocation can be zeroed. If the coefficients are zeroed, the quality ofthe enlargement may be somewhat compromised. To refine the magnitudes ofthe estimated coefficients, a cost function can also be used and can beapplied iteratively if desired.

[0045] In a different embodiment, to estimate sign and magnitudeinformation of the coefficients, a neural networks-based approach can beused. For example, a back-propagation learning method can be suitable.Other types of neural networks-based approaches can alternatively beused. A neural network based estimation technique allows a suitableneural network to learn a pattern of wavelet coefficients from the lowerlevels of the wavelet transform for each wavelet quarter separately, andfor each wavelet quarter, estimate the higher levels of coefficientsbased on the pattern. Neural network based estimation technique isapplied separately for each quarter as the pattern varies from quarterto quarter.

[0046]FIG. 7 is a diagrammatic illustration 110 of another technique forestimating wavelet coefficients. Wavelet quarters LL, HH, HD, and HVcarry different types of image information at each level. This method ofestimation uses this type of information to enhance the accuracy of theestimation. For example, it is known that HD quarters carry diagonalinformation, such as diagonal edges in the original image. Thus, in FIG.7, the wavelet coefficient X(1)_(HD3) is near a diagonal edge 112. Whena coefficient at level 3 near a diagonal edge is discovered (forexample, if its absolute value is above a threshold), correspondingpixel values at level 2 are estimated with greater accuracy because itis known that the diagonal information must also exist at level 2. InFIG. 7, therefore, the coefficients X(1 . . . 4)_(HD2) can be estimatedmore accurately since it is known that the diagonal edge 114 must bepreserved. Similarly, wavelet coefficients at level 1 at thecorresponding region of the image have values that can be accuratelyestimated, since they preserve the existence of the diagonal information116 that corresponds to the diagonal information in lower transformlevels. This image-preserving technique is image independent; an imageneed only have diagonal information. Some images, however, may carrymostly horizontal information; in these cases, the horizontalinformation can be assumed to be in other levels in the HH quarter andcan be used for more accurate estimations at higher HH levels, insimilar fashion to the diagonal information in HD quarters describedabove. Similarly, if an image carries vertical information, the verticalinformation can be assumed to be in other levels in the HV quarter andcan be used to for more accurate estimations at higher HV quadrantlevels, in similar fashion.

[0047] The information preserving technique described above can be usedin combination with other coefficient estimation techniques to enhancethe accuracy of the estimated coefficients. For example, themathematical modeling of wavelet coefficients at each level atcorresponding quarters can be used with information preserving inquarters.

[0048]FIG. 8 is a diagrammatic illustration 120 of another technique forestimating wavelet coefficients, using an averaging technique. Anaverage of neighboring coefficients is taken at one level and theaverage is used as a coefficient in the next level which corresponds tothose neighboring coefficients. This is typically most effective whenthe changes in the averaging locality are smooth. For example, in FIG.8, the coefficients X(1 . . . 16)_(HD1) at level 1 are known orestimated. The average of the four neighboring coefficientsX(11,12,15-16)_(HD1) yields a wavelet coefficient at level 2,X(4)_(HD2). Likewise, the wavelet coefficient X(3)_(HD2) can beestimated by averaging the four neighboring coefficientsX(9,10,13,14)_(HD1)., and the other coefficients X(1)_(HD2) andX(2)_(HD2) can be similarly estimated using X(1,2,5,6)_(HD1) andX(3,4,7,8)_(HD1), respectively. Corresponding values at level 3 andother levels can be estimated using the same technique.

[0049] It should be noted that the above-described techniques can becombined in various ways where appropriate to achieve greater accuracy.

[0050] Video Sequences

[0051] Video sequences are composed of a series of still images (frames)that are displayed to the user one at a time at a specified rate. Videosequences can take up a lot of memory or storage space when stored, andtherefore can be compressed so that they can be stored in smallerspaces. One way to compress video sequences is to remove frames,especially when a frame is positioned between two other frames in whichmost of the features in the frames remain constant and unchanged.Wavelet transform techniques of the present invention can be useful,when expanding a video sequence to its original (or to an even longer)length, to reconstitute the frames that have been lost duringcompression.

[0052]FIG. 9 is a diagrammatic illustration 130 of a sequence of videoimages and a technique of the present invention to estimate waveletcoefficients. In a video sequence, although two consecutive frames aregenerally similar, they carry unique information. One very accuratemethod to estimate the wavelet transformed coefficients is to use thetransformed coefficients from the past and future frames, as indicatedin FIG. 9. The coefficient 132 from the most immediate past frame 134can be compared to the coefficient 136 from the most immediate futureframe 138. If the signs of these past and future coefficients are thesame, then the sign for the estimated corresponding coefficient 140 inthe present frame 142 is estimated to be that same sign. If the signs ofthe past and future coefficients 132 and 136 have different signs, thenthe sign of the estimated present coefficient 140 is randomly selectedor estimated using the coefficients of the current frame. The absolutevalue of the coefficient 140 of the present frame 142 can be estimatedby averaging the absolute values of the coefficients 132 and 136 frompast and future frames 134 and 138 using wavelet extrapolation. Moresophisticated algorithms can also be used, such as Bayesian orProjections Onto Convex Sets (POCS). The result is that the coefficientsof the present frame have been estimated, and the present frame imagecan be reconstituted from these estimated coefficients even though thepresent frame had been deleted when in compressed form.

[0053] In addition, there may be a need for higher resolution videoframes. Since in many cases the resolution of a video sequence is low sothat the video can be displayed on a standard low-resolution television,there is a need to increase the resolution of the video images so thathigher-resolution video can be viewed on higher resolution devices, suchas computer screens and high definition television (HDTV).

[0054] When a single still image is evaluated, there is only spatialinformation that can be used in coefficient estimation. However, aseries of images in a video sequence additionally has a temporalcomponent, where images in the video appear to the user to move orchange over time. For a video sequence, therefore, temporal informationcan be used in wavelet coefficient estimation to increase the accuracyof the estimated coefficients. FIG. 10 is a block diagram 150illustrating the embodiment of the present invention for estimatingwavelet coefficients in a video sequence. The present frame 142 is inputto a wavelet transformer 152 to provide a number of wavelet transformsin the wavelet domain. Meanwhile, the most immediate past frame 134 isinput to a wavelet transformer 154 and the most immediate future frame138 is input to a wavelet transformer 156 to obtain one or more levelsof transforms of these frames in the wavelet domains. The wavelettransformers 152, 154, and 156 can all be the same transformer, used atdifferent times, or the same or different transformers usedsimultaneously.

[0055] In the estimator 158, HH, HD, and HV blocks having the same sizeas the present frame are estimated using the coefficients of the knownblocks (e.g., the LL block). To increase the accuracy of the estimation,not only are the known coefficients from transforms of the present frameused from transformer 152, but also the coefficients from transforms ofthe past and future frames are used from transformers 154 and 156, asdescribed above with reference to FIG. 9. Next, the Inverse DiscreteWavelet Transformer 160 applies the IDWT to the wavelet transformedframe, i.e., the present frame and estimated sub-bands. The output ofthe transformer 156 is an enlarged present frame 162 having twice thesize of the original frame. Alternatively, only the past frame or thefuture frame data can be used in the estimation additionally to thepresent frame data, for a less accurate estimation. Or, additional pastand/or future frame data can be used to increase the accuracy of theestimation.

[0056]FIG. 11 is a flow diagram illustrating a process 170 of thepresent invention for estimating wavelet coefficients from a series ofvideo images to provide enlarged video imagesor other enhanced videoimages. The process begins at 172, and in step 174 the level of therequired estimation in the present frame is determined. For example, inan iterative process, the coefficients in the LL quarter of the presentframe, followed by the HH quarter, HD quarter, and HV quarters, can beestimated, and a particular order of the coefficients in each quartercan be estimated; this step determines which coefficients to estimate.In next step 176, spatial estimation is used to estimate the requiredcoefficients. For example, any of the above-described methods of FIGS.5-8, or a combination thereof, can be used for coefficient estimation inthis step. In next step 178, a transformation of the most immediate pastframe is used to improve the spatial estimation numerically, where thetransformation of the past field is at a selected level. The firstiteration through the process, the past field transformation preferablyhas the same level as the level of transformation of the present field.In next step 180, the transformation of the past field is used toimprove the sign of the estimated coefficients. In step 182, atransformation of the most immediate future frame is used to improve thespatial estimation numerically, where the transformation of the futurefield is at a selected level. Preferably, the future fieldtransformation level is the same level as the level of transformation asthe past field used in step 178 b but can be different in otherembodiments. In step 184, the transformation of the future field is usedto improve the sign of the estimated coefficients. These steps aresimilar to steps 178 and 180. One example of steps 178 and 180 forestimating coefficients using data from past and future frames aredescribed above with reference to FIG. 9

[0057] In step 186, the process checks whether the accuracy of theestimated coefficients is sufficient; for example, if the rate of changeof improvement in estimation has dropped below a predetermined level.When the estimation reaches to such point, the process continues at step192, as described below. If the accuracy is not sufficient, then in step188, the process checks whether the transform levels have beenexhausted. But, after having a number of transforms, using moretransform levels doesn't help estimation, as one coefficient correspondsto too many coefficients in the level in which estimation is going to beperformed. Transform levels may be exhausted if all the transform levelsfrom which meaningful information could be extracted for past and futureframes have already been used in previous iterations of the presentprocess (see below). If the levels have been exhausted, the processcontinues to step 192. If they are not exhausted, then the processcontinues to step 190, in which one or more different levels oftransformation are selected for the past and future fields, and theprocess returns to step 178 so that steps 178, 180, 182, and 184 areperformed using the newly selected levels of transformation for past andfuture fields. This improves the numerical and sign accuracy of theestimation. For example, if the LL(2) quarter is being estimated and theLL(2) quarters for past and future fields were already used in the firstiteration, then when the process returns to step 178 for additionalaccuracy, the LL(1) and/or LL(3) quarters can also be used in the pastand future fields. These additional quarters can be used to estimate theLL(2) quarters of those fields using appropriate techniques, such as theapplicable techniques described above. With additionally accurate LL(2)quarters from past and future fields, the LL(2) quarter for the presentfield can be estimated with greater accuracy as well.

[0058] Once the accuracy of estimation is sufficient in step 186, and/orif the transform levels have been exhausted in step 188, the processcontinues to step 192, in which a check is made as to whethercoefficients need to be estimated at another level of transformation. Ifso, the process returns to step 174, and if not, the process is completeat 194. After step 194, an inverse wavelet transform can be applied tothe constructed wavelet transformed image, similar to the equivalentstep as described with respect to FIG. 10, to obtain the higherresolution video frame for the present frame.

[0059] Lost Data Recovery

[0060] There are many different application areas where waveletcoefficient estimation techniques can be used to enhance images or othertypes of data. One such area is the recovery of lost data in noisycommunication channels.

[0061]FIG. 12 is a diagrammatic illustration 250 of an example of therecovery of data lost in a communication channel. First, the locality ofthe lost data is determined in an existing wavelet transform tree. Then,the locality of the lost data is determined in lower and higher levelsof transform quadrants corresponding to the locality of the lost data.Finally, information at the locality in lower and higher levels is usedto estimate the lost data. For example, as shown in FIG. 12, a block ofdata 252 consisting of four wavelet coefficients is lost at level 2 inquarter HD, the high frequency diagonal quarter. This data is a set ofwavelet transformed coefficients at level 2. Since the coefficientscorresponding to this particular location are available at level 3(X(1)_(HD3)) and level 1 (X(1 . . . 16)_(HD1)), these known values canbe used to estimate the lost coefficients at level 2. In this example,one coefficient in a block corresponds to four coefficients in the nexthigher level block. So one known coefficient in level 3 can be used as a“seed” for estimating the four coefficients in level 2. In addition tothis, 16 known coefficients in level 1 also corresponds to these fourcoefficients and are employed for the estimation. If there is a lostcoefficient in a lower level block, a similar estimation method can beapplied to estimate the lost coefficient using the known coefficients inhigher level blocks. Any of the described estimation methods or asuitable combination can be used for estimation.

[0062] Data communication channels using a wavelet coefficientestimation technique of the present invention for lost data recovery mayencode data in such a way that more important data is less likely to belost. A significant aspect of this method is the determination of whichdata groups or packets are more important than other data.

[0063] Due to the properties of wavelet transform, if only lower levelblocks are available, images having a lower resolution than the originalimage resolution can still be constructed. For example, as shown in FIG.2 above, if LL(3), HH(3), HV(3), and HD(3) are known in the LL(2)quarter, then an image having {fraction (1/16)} of the originalresolution can be constructed. If all these level 3 quarters are knownas well as HH(2), HV(3), and HD(3), then an image having ¼ the originalresolution can be constructed. In other words, images of lowerresolution than the original image can still be constructed despite thelack of higher level blocks. Another property of wavelet transforms isthat the LL block of any level carries more of the character of theimage than any other block (HH, HV, or HD, i.e., “H(X)” to mean any ofthese high frequency quarters).

[0064]FIG. 13 is a diagrammatic illustration 270 of the blocks resultingfrom wavelet transforms, and indicates one example of defining therelative importance of the wavelet transformed blocks for lost datarecovery of the present invention. As shown, the LL block 272 is themost important. Following the LL block, the H(X) blocks 274 are listed.The H(X) blocks are ordered starting from the highest numbered level astheir effect on the image quality is more. Ordering is not definitiveand could be done in a different manner as well and the order shown inFIG. 13 is only exemplary.

[0065] In view of the above guidelines, the data communication channelshould preferably encode the wavelet transformed data in such a way asto ensure the error-free transmission of the most important blocks, suchas the LL blocks, or at least to minimize the chances of data lossoccurring for the more important blocks. Data loss may still occur.However, using the coefficient estimation techniques described above,wavelet coefficients can be obtained from the received data, and lostdata can be recovered and the image or video sequence can bereconstructed at the higher resolution (or other data constructreconstructed) with minimal or no artefacts.

[0066] Data compression is another important application area for theuse of wavelet coefficient estimation techniques. If a data lossrecovery algorithm of the present invention is operating on a datacommunication channel, the transmitter can deliberately send less data,omitting data from those areas or blocks which the coefficientestimation algorithm can estimate with the greatest accuracy. Forexample, some of the lower level H(X) blocks can be purposefullyomitted. This technique increases the compression of data withoutcompromising the quality of the transmitted images or video sequences.

[0067] Although the present invention has been described in accordancewith the embodiments shown, one of ordinary skill in the art willreadily recognize that there could be variations to the embodiments andthose variations would be within the spirit and scope of the presentinvention. For example, although the present invention is described inthe context of a frame being divided into four quadrants, or quarters,one of ordinary skill in the art recognizes that a frame could bedivided into any number of section and still be within the spirit andscope of the present invention. Accordingly, many modifications may bemade by one of ordinary skill in the art without departing from thespirit and scope of the appended claims.

What is claimed is:
 1. A method for creating an enlarged image from anoriginal image, the method comprising the steps of: (a) defining awavelet transformed image as a plurality of sections, wherein one of thesections is the original image; (b) estimating coefficients in estimatedsections of the wavelet transformed image, wherein the coefficients areestimated for each particular estimated section using informationobtained from that section; and (c) performing an inverse wavelettransform on the wavelet transformed image, to obtain the enlargedimage.
 2. The method of claim 1 wherein the plurality of sectionscomprises four quarters.
 3. The method of claim 2 wherein a Low-Low (LL)quarter of the wavelet transformed image contains the original image. 4.The method of claim 3 wherein the second, third, and fourth quarters ofthe wavelet transformed image are a High-Horizontal (HH) quarter, aHigh-Diagonal (HD) quarter, and a High-Vertical (HV) quarter,respectively.
 5. The method of claim 1 further comprising repeatingsteps a, b, and c to create an enlarged image with a greater resolution.6. The method of claim 5 wherein in the greater resolution enlargedimage is defined as the first quarter in the wavelet transform.
 7. Themethod of claim 2 wherein the estimating step (b) includes the step of(b1) estimating absolute values of the coefficients of the estimatedquarters by finding a correlation of wavelet coefficients at differenttransform levels and extrapolating the correlation to higher levelsusing a pattern.
 8. The method of claim 7 wherein the estimating step b1includes the step of (b2) estimating signs of the coefficients of theestimated quarters by a probability or statistical technique.
 9. Themethod of claim 2 wherein if the estimating of a non-zero coefficientsis not accurate to a predetermined degree, at least some of thecoefficients are zeroed.
 10. The method of claim 2 wherein theestimating step (b) includes using a neural network based estimationtechnique.
 11. The method of claim 10 wherein the neural network basedestimation technique allows a suitable neural network to learn a patternof wavelet coefficients from the lower levels of the wavelet transformfor each wavelet quarter separately, and for each wavelet quarter,estimate the higher levels of coefficients based on the pattern.
 12. Themethod as recited in claim 2 wherein the estimating step (b) includes:(b1) examining spatial features in particular lower level quarters ofthe at least one lower level wavelet transform of the original image;and (b2) estimating coefficients in the estimated quarters thatcorrespond to the particular lower level quarters, the coefficientshaving spatial features similar to the examined spatial features. 13.The method as recited in claim 12 wherein the examining step (b1) andestimating step (b2) include: using lower level HH quarters forestimating horizontal features in the coefficients; using lower level HVquarters for estimating vertical features in the coefficients; and usinglower level HD quarters for estimating diagonal features in thecoefficients.
 14. The method of claim 2 wherein the estimating step (b)includes the step of (b1) averaging neighboring coefficients at onelevel of the wavelet transform of the original image and using theaverage as an estimated coefficient in another level of the wavelettransform, the estimated coefficient corresponding to the neighboringcoefficients.
 15. A method for creating high resolution video frames ina video sequence, the method comprising the steps of: (a) defining awavelet transformed frame as a plurality of sections, wherein one of thesections is a present video frame; (b) estimating coefficients inestimated sections of the wavelet transformed frame, wherein thecoefficients are estimated for each particular estimated section usinginformation from a lower level wavelet transform of a correspondingsection of a past or future frame of the video sequence; and (c)performing an inverse wavelet transform on the wavelet transformedframe, the wavelet transformed frame including the present video frameand the estimated sections, to obtain a high resolution video framehaving a greater resolution than the present video frame.
 16. The methodof claim 15 wherein the plurality of sections are quarters of the frame.17. The method of claim 16 wherein the coefficients are estimated foreach particular estimated quarter also using information obtained fromat least one lower level wavelet transform of the present video framefor that particular estimated quarter.
 18. The method of claim 16wherein the coefficients estimated are non-zero coefficients.
 19. Themethod of claim 16 wherein the estimating step (b) includes the step of(b1) finding a correlation of wavelet coefficients at differenttransform levels and extrapolating the correlation to higher levelsusing a pattern.
 20. The method of claim 19 wherein the estimation isimproved in accuracy by using the same level of transform in the pastframe of the video sequence and using the same level of transform in thefuture frame of the video sequence.
 21. The method of claim 20 whereinthe past frame being just previous to the present frame and the futureframe being just after the present frame in the video sequence.
 22. Themethod of claim 15 further comprising the step of (d) improving theaccuracy of the estimate of coefficients by using additional levels inwavelet transform sections in the past frame and future frame of thevideo sequence.
 23. The method of claim 15 wherein the estimating step(b) includes the step of (b1) averaging the absolute values of thecoefficients from the past and future frames using waveletextrapolation.
 24. A method for recovering data lost from original dataduring transmission of the original data in a communication channel, themethod comprising the steps of: (a) determining the locality of the lostdata in a wavelet transform tree, the wavelet transform tree having beencreated from the original data before transmission; (b) determining thelocality of the lost data in lower and higher levels of the wavelettransform tree and in wavelet transform quadrants corresponding to thelocality of the lost data; and (c) using information at the locality inthe lower and higher levels of the wavelet transform tree to estimatethe lost data.
 25. The method of claim 24 further comprising the step of(d) reducing the amount of lost data during transmission in thecommunication channel, wherein the reduction in the amount of lost dataincludes ordering the quadrants in all the levels of the transform in alist of descending priority, and using higher priority communicationchannels to transmit higher priority data.
 26. The method of claim 25wherein Low-Low (LL) quadrants are considered higher priority than highfrequency quadrants of High-Horizontal (HH), High-Vertical (HV), andHigh-Diagonal (HD).
 27. The method of claim 26 wherein the original dataincludes an image or video sequence.
 28. A system for creating anenlarged image from an original image comprising: wavelet transformerfor defining a wavelet transformed image as a plurality of sections,wherein one of the sections is the original image; an estimator forestimating coefficients in estimated sections of the wavelet transformedimage, wherein the coefficients are estimated for each particularestimated section using information obtained from that section; and aninverse transform mechanism for performing an inverse wavelet transformon the wavelet transformed image to obtain the enlarged image.
 29. Thesystem of claim 28 wherein the plurality of sections comprises fourquarters.
 30. The system of claim 29 wherein a Low-Low (LL) quarter ofthe wavelet transformed image contains the original image.
 31. Thesystem of claim 30 wherein the second, third, and fourth quarters of thewavelet transformed image are a High-Horizontal (HH) quarter, aHigh-Diagonal (HD) quarter, and a High-Vertical (HV) quarter,respectively.
 32. The system of claim 28 further comprising utilizingthe wavelet transformer, estimator and inverse transform mechanismrepeatedly.
 33. The system of claim 32 wherein in the greater resolutionenlarged image is defined as the first quarter in the wavelet transform.34. The system of claim 32 wherein the estimator includes means forestimating absolute values of the coefficients of the estimated quartersby finding a correlation of wavelet coefficients at different transformlevels and means for extrapolating the correlation to higher levelsusing a pattern.
 35. The system of claim 34 wherein the estimatorincludes means for estimating signs of the coefficients of the estimatedquarters by a probability or statistical technique.
 36. The system ofclaim 29 wherein if the estimating of the coefficients is not accurateto a predetermined degree, at least some of the coefficients are zeroed.37. The system of claim 29 wherein the estimator includes using a neuralnetwork based estimation technique.
 38. The system of claim 37 whereinthe neural network based estimation technique allows a suitable neuralnetwork to learn a pattern of wavelet coefficients from the lower levelsof the wavelet transform for each wavelet quarter separately, and foreach wavelet quarter, estimate the higher levels of coefficients basedon the.
 39. The system of claim 29 wherein the estimator includes: meansfor examining spatial features in particular lower level quarters of theat least one lower level wavelet transform of the original image; andmeans for estimating coefficients in the estimated quarters thatcorrespond to the particular lower level quarters, the coefficientshaving spatial features similar to the examined spatial features. 40.The system as recited in claim 39 wherein the examining means andestimating means include: means for using lower level HH quarters forestimating horizontal features in the coefficients; means for usinglower level HV quarters for estimating vertical features in thecoefficients; and means for using lower level HD quarters for estimatingdiagonal features in the coefficients.
 41. The system of claim 29wherein the estimator includes means for averaging neighboringcoefficients at one level of the wavelet transform of the original imageand using the average as an estimated coefficient in another level ofthe wavelet transform, the estimated coefficient corresponding to theneighboring coefficients.
 42. A system for creating high resolutionvideo frames in a video sequence, the system comprising: a wavelettransformer for defining a wavelet transformed frame as a plurality ofsections, wherein one of the sections is a present video frame; anestimator estimating coefficients in estimated sections of the wavelettransformed frame, wherein the coefficients are estimated for eachparticular estimated section using information from a lower levelwavelet transform of a corresponding section of a past or future frameof the video sequence; and an inverse transform mechanism for performingan inverse wavelet transform on the wavelet transformed frame, thewavelet transformed frame including the present video frame and theestimated sections, to obtain a high resolution video frame having agreater resolution than the present video frame.
 43. The system of claim42 wherein the plurality of sections are quarters of the frame.
 44. Thesystem of claim 43 wherein the coefficients are estimated for eachparticular estimated quarter also using information obtained from atleast one lower level wavelet transform of the present video frame forthat particular estimated quarter.
 45. The system of claim 44 whereinthe coefficients estimated are non-zero coefficients.
 46. The system ofclaim 44 wherein the estimating step (b) includes the step of (b1)finding a correlation of wavelet coefficients at different transformlevels and extrapolating the correlation to higher levels using apattern.
 47. The system of claim 46 wherein the estimation is improvedin accuracy by using the same level of transform in the past frame ofthe video sequence and using the same level of transform in the futureframe of the video sequence.
 48. The system of claim 47 wherein the pastframe being just previous to the present frame and the future framebeing just after the present frame in the video sequence.
 49. The systemof claim 42 further comprising means for improving the accuracy of theestimate of coefficients by using additional levels in wavelet transformsections in the past frame and future frame of the video sequence. 50.The system of claim 42 wherein the estimator includes means foraveraging the absolute values of the coefficients from the past andfuture frames using wavelet extrapolation.
 51. A system for recoveringdata lost from original data during transmission of the original data ina communication channel, the system comprising: means for determiningthe locality of the lost data in a wavelet transform tree, the wavelettransform tree having been created from the original data beforetransmission; means for determining the locality of the lost data inlower and higher levels of the wavelet transform tree and in wavelettransform sections corresponding to the locality of the lost data; andmeans for using information at the locality in the lower and higherlevels of the wavelet transform tree to estimate the lost data.
 52. Thesystem of claim 51 wherein the wavelet transformed sections comprisewavelet transformed quadrants.
 53. The system of claim 52 furthercomprising reducing the amount of lost data during transmission in thecommunication channel, wherein the reduction in the amount of lost dataincludes ordering the quadrants in all the levels of the transform in alist of descending priority, and using higher priority communicationchannels to transmit higher priority data.
 54. The method of claim 53wherein Low-Low (LL) quadrants are considered higher priority than highfrequency quadrants of High-Horizontal (HH), High-Vertical (HV), andHigh-Diagonal (HD).
 55. The method of claim 54 wherein the original dataincludes an image or video sequence.
 56. A computer readable mediumcontaining program instructions for creating an enlarged image from anoriginal image, the program instructions: (a) defining a wavelettransformed image as a plurality of sections, wherein one of thesections is the original image; (b) estimating coefficients in estimatedsections of the wavelet transformed image, wherein the coefficients areestimated for each particular estimated section using informationobtained from that section; and (c) performing an inverse wavelettransform on the wavelet transformed image, to obtain the enlargedimage.
 57. A computer readable medium containing program instructionsfor creating high resolution video frames in a video sequence, theprogram instructions for: (a) defining a wavelet transformed frame as aplurality of sections, wherein one of the sections is a present videoframe; (b) estimating coefficients in estimated sections of the wavelettransformed frame, wherein the coefficients are estimated for eachparticular estimated section using information from a lower levelwavelet transform of a corresponding section of a past or future frameof the video sequence; and (c) performing an inverse wavelet transformon the wavelet transformed frame, the wavelet transformed frameincluding the present video frame and the estimated sections, to obtaina high resolution video frame having a greater resolution than thepresent video frame.
 58. A computer readable medium for recovering datalost from original data during transmission of the original data in acommunication channel, the program instructions for: (a) determining thelocality of the lost data in a wavelet transform tree, the wavelettransform tree having been created from the original data beforetransmission; (b) determining the locality of the lost data in lower andhigher levels of the wavelet transform tree and in wavelet transformquadrants corresponding to the locality of the lost data; and (c) usinginformation at the locality in the lower and higher levels of thewavelet transform tree to estimate the lost data.